On the Opportunities and Risks of Foundation Models

AI is undergoing a paradigm shift with the rise of models (e.g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks. (Abstractより)

「多様なデータで訓練され、広い範囲の下流タスクに適用できるモデルの出現によるパラダイムシフト」

We call these models foundation models to underscore their critically central yet incomplete character.

foundation model提唱

deep learning, pretraining is the dominant approach to transfer learning: a model is trained on a surrogate task (often just as a means to an end) and then adapted to the downstream task of interest via fine-tuning. (1.1)